Skip to content

VMware→KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes and Linstor root/data volumes#13656

Open
andrijapanicsb wants to merge 25 commits into
apache:mainfrom
andrijapanicsb:kvm-rbd-vmware-migration
Open

Conversation

@andrijapanicsb

@andrijapanicsb andrijapanicsb commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Description

This PR makes Ceph/RBD and Linstor first-class targets for KVM VM onboarding and adds an operator-driven warm VMware→KVM migration path. It bundles the following (Ceph/RBD plus Linstor as block-storage destinations):

1. VMware CBT → native RBD/qcow2 warm migration to KVM

An operator-driven, warm VMware-to-KVM migration built on VDDK and VMware Changed Block Tracking (CBT). It maintains a source-equivalent KVM-side replica kept current through repeated delta/sync cycles, then finalizes the guest with virt-v2v in-place at cutover — bounding downtime to the cutover window instead of a full cold copy.

  • Targets both filesystem (qcow2) and native Ceph/RBD (raw image) primary storage.
  • Full lifecycle as async operations with persisted migration state (migration / cycle / disk records), preflight validation, cancel/cleanup, and Import/Export UI integration.
  • New admin API commands: startVmwareCbtMigration, syncVmwareCbtMigration, cutoverVmwareCbtMigration, cancelVmwareCbtMigration, deleteVmwareCbtMigration, listVmwareCbtMigrations, checkVmwareCbtMigrationPrerequisites.
  • Server orchestration in VmwareCbtMigrationManagerImpl (+ VmwareCbtMigrationServiceImpl, cutover policy), backed by new VOs/DAOs and a DB schema migration.
  • KVM agent: libvirt CBT command wrappers (prepare / sync / cutover / RBD-probe) driving VDDK + qemu + virt-v2v, plus agent commands/answers and a sync-plan.
  • UI: new VMware CBT Migrations view and additions to Import/Manage Instances under Tools.
  • Docs: docs/vmware-cbt/ (architecture + README) and docs/vmware-cbt-migration.md.

2. Direct VDDK VMware import into Ceph/RBD — the cold (one-shot) path

The cold counterpart to the warm CBT migration in (1): extends the existing
VDDK + virt-v2v VMware→KVM import — which previously landed only on NFS/qcow2
(direct, or via a staged file) — so the converted disk is written straight into an
RBD pool as a raw image
, with in-place virt-v2v finalization and no intermediate
qcow2-on-NFS staging copy. (The warm path in (1) reuses this same full-disk copy for its
initial sync.) Adds host-capability detection for qemu-img RBD support, RBD qemu-copy,
virt-v2v in-place, and direct-RBD import — advertised via ReadyCommand and reconciled
by the agent manager.

3. importVm adoption of existing RBD (Ceph) root volumes and Linstor root + data volumes

Extends the KVM unmanaged-import / manage-volume flow so an existing raw RBD image can be adopted directly as a VM ROOT volume. Adds RBD-aware format handling in VolumeOrchestrator (RBD ⇒ RAW instead of the hardcoded QCOW2) and the RBD branch in the libvirt volume-inspection path (LibvirtCheckVolumeCommandWrapper), so CheckVolumeCommand succeeds on RBD instead of failing with "Disk not found or is invalid".

Also extends the same adoption path to Linstor (DRBD) primary storage, for both ROOT and DATA volumes: an externally prepared Linstor volume (e.g. landed by a migration/conversion tool) can be adopted without any data copy. Linstor volumes are raw local block devices, so volume inspection runs qemu-img against the /dev/drbd/… device path (LibvirtCheckVolumeCommandWrapper for ROOT, LibvirtGetVolumesOnStorageCommandWrapper + the volume import/unmanage gate for DATA), and VolumeOrchestrator records RAW. Because a host-local qemu-img file lock cannot see a DRBD resource that is Primary/open on another node (unlike RBD's cluster-wide exclusive lock), adoption additionally consults Linstor's cluster-wide InUse state — new KVMStoragePool.getVolumeInUseNode() capability (default no-op), answered via the existing LinstorUtil.isResourceInUse — and refuses a volume attached to a running VM anywhere in the cluster. Volumes are adopted under their existing resource names (Linstor cannot rename; 48-char limit).

4. Consolidate KVM host-capability probe names across the RBD features

Because features (1) and (2) introduced host-capability keys independently, this commit unifies them under feature-neutral names that describe what is probed, and drops a redundant alias (host.rbd.qemu.copy.support, identical to host.qemu.rbd.support) and its delegate.

5. Linstor (DRBD) primary storage as a migration destination

Extends all three VMware→KVM paths above — staged import, direct VDDK import, and warm CBT migration — so the converted disks can also land on Linstor primary storage, not just Ceph/RBD. A generic RAW_BLOCK_DEVICE target type covers "write RAW into a host-local block device provided by the storage adaptor", so the same code paths serve Linstor (and lay groundwork for other block backends).

  • The KVM agent pre-creates each Linstor resource at source capacity (unlike RBD, qemu-img cannot create a DRBD device), copies with qemu-img convert -n — or nbdcopy when the host has it — into the local /dev/drbd/… device, and finalizes with virt-v2v-in-place fed a <disk type='block'> domain XML (no qemu-nbd bridge needed for a local device).
  • Warm CBT delta cycles stream each changed extent from the nbdkit/VDDK source window straight into the same device window (qemu-img convert -n -S 0), avoiding a temp-file round-trip.
  • Server orchestration adds Linstor to the staged / direct-VDDK / CBT destination pool types and validates that the conversion host is a LINSTOR satellite connected to the pool.
  • LINSTOR resource-name limits (48 chars, no rename) are respected with short deterministic names; imported volumes keep the name as their recorded path.
  • Also drops an unsupported -O flag from the virt-v2v-in-place invocation (fixes in-place finalization on virt-v2v < 2.5, e.g. Ubuntu 24.04) — which benefits the RBD path too.

6. Data-plane efficiency and correctness for the agent-driven copies

These apply to the agent's own nbdkit/VDDK full-disk copies (cold direct import and CBT initial full sync); they do not touch the staged virt-v2v conversion path.

  • VDDK NBD compression — the nbdkit VDDK plugin now runs with compression=<value> (new vddk.nbd.compression agent property, default fastlz; none/zlib/fastlz/skipz). On datastores that don't expose block allocation to VDDK, the full nominal disk size otherwise crosses the network; fastlz compresses both zero regions and real data on the ESXi side at low CPU cost (measured on a 30 GiB disk: 32.4 GB → 2.65 GB over the wire).
  • Multi-connection nbdcopy for RBD full copies — cold-direct-to-RBD and the CBT initial full sync to RBD now copy with nbdcopy over a localhost qemu-nbd bridge (pre-create the raw image, serve it locally, copy NBD-to-NBD) instead of a single-stream qemu-img convert, falling back to qemu-img convert when nbdcopy is unavailable. The Linstor block-device path already used nbdcopy directly; CBT delta cycles are unchanged (a range copier is required there).
  • Thin-safe zero handling — copies into a pre-created block-device target skip writing zero blocks (--target-is-zero / --destination-is-zero) only when the backend guarantees a freshly created volume reads back as zeros, via a new KVMStoragePool.isVolumeZeroInitialized() capability (Linstor answers it from the provider kind — thin providers only). On backends that do not guarantee it (e.g. LVM-thick) the zeros are written, so no stale data from a previously deleted volume leaks into the unwritten regions. qemu-img's --target-is-zero support is also probed directly (newer qemu dropped it from --help while still supporting it).

Types of changes

  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Bug fix
  • Enhancement

Feature/Enhancement Scale

  • Major
  • Minor

How Has This Been Tested?

  • Unit tests for all new and touched wrappers/managers/orchestration:
    LibvirtImportConvertedInstanceCommandWrapperTest, LibvirtConvertInstanceCommandWrapperTest,
    LibvirtCheckConvertInstanceCommandWrapperTest, LibvirtCheckVolumeCommandWrapperTest,
    the LibvirtVmwareCbt* suites, UnmanagedVMsManagerImplTest,
    VolumeImportUnmanageManagerImplTest, the server-side VmwareCbt* suites,
    and VolumeOrchestratorTest — all passing.

  • Manual/lab validation of warm CBT migration and direct-RBD import against a Ceph/RBD KVM cluster.

  • Linstor destination validated byte-identical on a live 3-node LINSTOR/DRBD lab against a real VMware source (cold direct-VDDK and warm CBT; qemu-img compare = images identical), and end-to-end through a live importVm that produced a Stopped CloudStack VM with its root and data disks as Linstor volumes backed by DRBD resources UpToDate on all three nodes. New Linstor unit tests across the destination gate lists, host selection, the convert/import/CBT wrappers, listPhysicalDisks, and the block-device delta/nbdcopy paths — all passing.

  • Linstor adoption (root + data) covered by new unit tests: CheckVolume inspection via the /dev/drbd device path (never an rbd: URI), the in-use-on-another-node refusal, and the Linstor ⇒ RAW format resolution — all passing; live adopt-and-boot validation on the 3-node LINSTOR lab to follow.

Operator-driven warm VMware-to-KVM migration built on VDDK and VMware
Changed Block Tracking (CBT). Keeps a source-equivalent KVM-side replica
current through delta cycles, then finalizes with virt-v2v in-place at
cutover. Supports both filesystem (qcow2) and native Ceph/RBD (raw image)
primary storage targets, with preflight validation, async lifecycle
operations, persisted migration state, and Import/Export UI integration.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Extends the existing VDDK + virt-v2v VMware-to-KVM import path so the
converted disk can be written straight into an RBD pool as a raw image,
with in-place virt-v2v finalization. Adds host-capability detection for
qemu-img RBD support, RBD qemu copy, virt-v2v in-place, and direct RBD
import, advertised via ReadyCommand and reconciled by the agent manager.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Extends the KVM unmanaged import / manage-volume flow so an existing raw
RBD image can be adopted directly as a VM root volume, including the
RBD-format check in the volume orchestrator and the libvirt volume
inspection path.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Unifies the host-capability keys that the CBT warm-migration and VDDK
direct-RBD-import features introduced independently under different
names, using feature-neutral names that describe what is probed:
  host.vmware.cbt.support                     -> host.vddk.blockcopy.support
  host.vmware.cbt.in.place.finalization.support -> host.vddk.blockcopy.inplace.finalization.support
  host.vmware.cbt.rbd.support                 -> host.vddk.blockcopy.rbd.support
  host.qemu.img.rbd.support                   -> host.qemu.rbd.support
  host.virt.v2v.inplace.support / host.virtv2v.in.place.version -> host.virtv2v.inplace.*
Drops the redundant host.rbd.qemu.copy.support alias (identical to
host.qemu.rbd.support) and its hostSupportsRbdQemuCopy() delegate.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 37.48240% with 3109 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.76%. Comparing base (fc83241) to head (a7d460d).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
.../wrapper/LibvirtConvertInstanceCommandWrapper.java 12.26% 368 Missing and 4 partials ⚠️
.../vmware/manager/VmwareCbtMigrationServiceImpl.java 0.28% 354 Missing and 1 partial ⚠️
...c/main/java/com/cloud/vm/VmwareCbtMigrationVO.java 9.45% 201 Missing ⚠️
...ervisor/kvm/resource/LibvirtComputingResource.java 8.43% 150 Missing and 2 partials ⚠️
...i/command/admin/vm/StartVmwareCbtMigrationCmd.java 0.00% 140 Missing ⚠️
...wrapper/LibvirtVmwareCbtCutoverCommandWrapper.java 77.92% 82 Missing and 37 partials ⚠️
.../apache/cloudstack/vm/UnmanagedVMsManagerImpl.java 56.06% 96 Missing and 20 partials ⚠️
...wrapper/LibvirtVmwareCbtPrepareCommandWrapper.java 64.50% 85 Missing and 30 partials ⚠️
...ce/wrapper/LibvirtVmwareCbtSyncCommandWrapper.java 67.06% 82 Missing and 28 partials ⚠️
.../response/VmwareCbtMigrationPreflightResponse.java 0.00% 97 Missing ⚠️
... and 61 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #13656      +/-   ##
============================================
+ Coverage     19.64%   19.76%   +0.11%     
- Complexity    19789    20192     +403     
============================================
  Files          6368     6420      +52     
  Lines        575109   582201    +7092     
  Branches      70371    71221     +850     
============================================
+ Hits         112994   115060    +2066     
- Misses       449829   454652    +4823     
- Partials      12286    12489     +203     
Flag Coverage Δ
uitests 3.39% <ø> (-0.02%) ⬇️
unittests 21.04% <37.48%> (+0.12%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@blueorangutan package kvm

@blueorangutan

Copy link
Copy Markdown

@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with kvm SystemVM template(s). I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18626

@andrijapanicsb andrijapanicsb added this to the 4.24.0 milestone Jul 21, 2026
@andrijapanicsb andrijapanicsb changed the title KVM/VMware: warm CBT migration and direct VDDK import into Ceph/RBD, plus importVm adoption of existing RBD root volumes KVM/VMware: warm/change tracking migration and direct VDDK import into Ceph/RBD, plus importVm adoption of existing RBD root volumes Jul 21, 2026
andrijapanicsb and others added 4 commits July 21, 2026 06:58
…isk import

The "Manage Instances" source dropdown offered "Import QCOW2 image from
Local/Shared Storage", but importVm now adopts an existing ROOT disk from
Filesystem, NetworkFilesystem, SharedMountPoint and RBD (raw) pools, so the
QCOW2-only wording is misleading. Rename both entries and their wizard
title/description to "Import Instance using existing ROOT disk from
Local/Shared Storage", across the en, pt_BR and te locales.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
docs/vmware-cbt/architecture.md and docs/vmware-cbt/README.md were added
without the Apache license header, which failed the RAT license check and the
pre-commit insert-license hook. Add the standard ASF header (identical to the
one already present in docs/vmware-cbt-migration.md).

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
…ey names

State Apache CloudStack 4.24.0.0 as the target release (schema delivered via the
4.24.0.0 upgrade path, with a note that the DDL currently rides in
schema-42210to42300.sql pending the main version bump), and replace the
pre-consolidation host-capability key names with the current
host.vddk.blockcopy.* / host.virtv2v.inplace.version names.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Staged VMware-to-KVM imports (virt-v2v conversion to an NFS temporary
location followed by a copy to the destination pool) can now target
Linstor primary storage:

- Linstor added to the staged-conversion destination pool types, so
  destination pool discovery considers cluster- and zone-wide Linstor
  pools alongside NFS and RBD.
- The import host is validated to have access to each Linstor
  destination pool (it must be a LINSTOR satellite connected to the
  pool); the RBD qemu-copy capability check is preserved and the
  validation renamed to validateStagedImportHostSupport.
- The KVM-side converted-disk move already dispatches by pool type to
  LinstorStorageAdaptor.copyPhysicalDisk, which spawns the resource
  from the pool's resource group and qemu-img converts the qcow2 into
  the DRBD block device; converted-disk metadata for Linstor pools now
  reports the pool UUID and volume name instead of parsing NFS mounts
  (Linstor pools have no local mount path).

The imported volume keeps the copy-time UUID as its path; the Linstor
driver addresses existing volumes by path, so later volume lifecycle
operations (attach, resize, snapshot, delete) work unchanged.
A standalone design/validation document covering the Linstor-destination
portion of this series, written so LINSTOR/DRBD maintainers can review
the storage-side behavior: the RAW_BLOCK_DEVICE target model,
pre-created resources with qemu-img convert -n / nbdcopy, in-place
finalization on the local DRBD device, resource-naming constraints,
delta-copy semantics, host prerequisites, the support matrix, and the
lab/end-to-end validation performed.
@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18633

@andrijapanicsb andrijapanicsb self-assigned this Jul 21, 2026
import static com.cloud.host.Host.HOST_VIRTV2V_INPLACE_SUPPORT;
import static com.cloud.host.Host.HOST_VIRTV2V_INPLACE_VERSION;
import static com.cloud.host.Host.HOST_VIRTV2V_VERSION;
import static com.cloud.host.Host.HOST_VDDK_BLOCKCOPY_INPLACE_FINALIZATION_SUPPORT;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: these new static imports (HOST_VDDK_BLOCKCOPY_*) break the existing alphabetical import order (they land after HOST_VDDK_VERSION instead of before HOST_VDDK_LIB_DIR).

import java.util.List;
import java.util.Map;

import org.apache.commons.collections.CollectionUtils;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this imports the legacy org.apache.commons.collections.CollectionUtils (also in LibvirtVmwareCbtCleanupCommandWrapper.java, LibvirtVmwareCbtCutoverCommandWrapper.java, LibvirtVmwareCbtPrepareCommandWrapper.java), while other new files in this PR use commons.collections4. Worth picking one.

Comment thread ui/public/locales/en.json
"message.desc.import.ext.kvm.wizard": "Import libvirt domain from External KVM Host not managed by CloudStack",
"message.desc.import.local.kvm.wizard": "Import QCOW2 image from Local Storage of selected KVM Host",
"message.desc.import.shared.kvm.wizard": "Import QCOW2 image from selected Primary Storage Pool",
"message.desc.import.local.kvm.wizard": "Import an Instance using an existing ROOT disk from the Local Storage of the selected KVM host",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this changes "selected KVM Host" to "selected KVM host" (lowercase) and the sibling string below drops "Storage Pool" to "Storage pool", a casing regression from the previous strings.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Labeling is SO incosistent (plsm check that file 5000 times "host" and another 5000 times Host - I'm lazy - but feel free to change it if you want

I'm more interested in actual testing the value of the PR than semantically

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@rp- — this PR now also adds Linstor as a destination for the
VMware→KVM import/migration paths (cold staged, cold direct-VDDK, and warm
CBT), on top of the Ceph/RBD support. I'd really appreciate a review of the
Linstor/DRBD side from you.

To make that easy there's a standalone write-up committed in the PR:
docs/vmware-linstor-migration.md

The Linstor-side design points I'd most like sanity-checked:

  1. Pre-created resources + qemu-img convert -n — since qemu can't create
    a DRBD device, the agent pre-creates each resource at source capacity via
    LinstorStorageAdaptor and writes with no-create mode (relying on DRBD's
    size round-up being >= source).
  2. Single-Primary / auto-promote — the conversion host opens the device
    exclusively during the copy and the in-place virt-v2v finalization; devices
    return to Secondary afterwards. Any pitfalls we should guard against?
  3. 48-char resource names, no rename — short deterministic names
    (<uuid>-d%02d for direct imports, cbt-<mig8>-<diskId> for warm CBT);
    imported volumes keep the name as their recorded volume path.
    listPhysicalDisks is now implemented (RD list filtered by the pool's
    resource group).
  4. Diskful-satellite preference for the conversion host is recommended in
    the docs but not enforced — should it be?

Validated on a 3-node LINSTOR/DRBD 9 lab: byte-identical disk copies
(qemu-img compare) for both cold and warm paths, plus a full end-to-end
importVm producing a CloudStack VM with Linstor-backed volumes, UpToDate on
all replicas. Happy to adjust anything on the Linstor side — thanks!

@wido

wido commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Great to see this! I can't test it (don't have any env or even VMware), but a review will be though. It's a big PR with a lot of lines to check.

I assume you tested this? Can it break any existing things?

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

Great to see this! I can't test it (don't have any env or even VMware), but a review will be though. It's a big PR with a lot of lines to check.

I assume you tested this? Can it break any existing things?

Tested "offline" but also being tested "more" properly as well.

Smoke tests etc - comes later - should not break anything
(existing related stuff such as pure VDDK migration is also being tested, of course etc - but other/general code has not been touched - it's purelly to add RBD/Linstore as supported destination storages, plus some unrelated magic wink)

EL9-family distributions (Rocky/RHEL/Alma 9) install virt-v2v-in-place in
/usr/libexec, outside $PATH. Probing the bare binary name therefore reported
host.virtv2v.inplace.support=false, which cascaded to
host.vddk.blockcopy.inplace.finalization.support and
host.vddk.rbd.direct.import.support and silently disabled the direct-VDDK and
warm-CBT block-device import paths on the most common production platform.

Resolve the executable via LibvirtComputingResource.getVirtV2vInPlaceBinary()
($PATH first, then /usr/libexec) and use the resolved binary in the capability
probe, the version detection, and every finalization invocation (direct
convert and CBT cutover).
Direct VDDK import into an RBD pool copied each disk, then finalized with
virt-v2v-in-place fed a domain XML whose disks were native
<disk type='network' protocol='rbd'> sources with a cephx <auth> element
referencing a libvirt secret. virt-v2v's '-i libvirtxml' input runs without a
libvirt connection, so the secret could not be resolved and libguestfs
attached no drives, failing with 'you must call guestfs_add_drive before
guestfs_launch' (observed on EL9).

Serve each RBD image over a temporary localhost qemu-nbd bridge and point the
finalization XML at the NBD endpoints instead - the same approach the CBT
cutover already uses - so no secret needs resolving. The bridges are torn
down on exit.
Copying a converted disk into a freshly created thin block-device target
(Linstor/DRBD) fully allocated the volume: qemu-img convert and nbdcopy do
not assume a block-device destination is zeroed, so they wrote out every zero
block. A 30 GiB volume holding ~2 GiB of data became 99.99% allocated on all
replicas, exhausting the thin pool.

Pass -n --target-is-zero to qemu-img convert and --destination-is-zero to
nbdcopy for pre-created block-device targets. Also fix the QemuImg
--target-is-zero capability detection: newer qemu (10.x on EL9) no longer
lists the flag in 'qemu-img --help' while still supporting it, so probe the
convert option directly - this is also why the Linstor adaptor's existing
zero-skip (copyPhysicalDisk) silently stopped engaging.
…deletes

When a CBT migration is cancelled, its initial/delta sync data-plane copy is
not aborted by the cancel itself, so it can still be writing into - and
holding open - the very target volumes the cleanup then tries to delete. The
surviving copy also keeps burning VDDK/network/storage on a dead migration.

Before deleting the marker-guarded target images/volumes, kill the in-flight
copy processes whose command line references this migration's marker (matches
the nbdcopy / qemu-img convert / qemu-io processes; the nbdkit --run parent
exits with its child), then delete with a short retry to ride out the brief
window where the device is still held open (DRBD demotion, qemu teardown).
…BT states

- importVm (VMware to KVM): validate the data-disk offering mappings right
  after the source instance is discovered, before any conversion starts. The
  same count check previously ran only at import time, after every byte of
  the source disks had already been copied and converted (~20 wasted minutes
  per attempt in lab testing), and a validation failure at that late stage
  also orphaned the already-copied direct-import volumes.

- startVmwareCbtMigration: same pre-flight against the discovered source
  disks, so a missing datadiskofferinglist fails before the initial full
  sync instead of at cutover import.

- markMigrationFailed: do not overwrite a terminal migration state - a cancel
  racing with an in-flight sync previously flipped Cancelled to Failed when
  the aborted sync's failure landed afterwards.
…py for RBD)

Two data-plane efficiency improvements for the agent's own nbdkit full-disk
copies (cold direct import and CBT initial full sync). Neither touches the
staged virt-v2v conversion path, which drives its own VDDK input.

- VDDK NBD compression: the nbdkit vddk plugin now runs with
  compression=<value> (new agent.properties key vddk.nbd.compression,
  default fastlz; none/zlib/fastlz/skipz). In extent-less setups (e.g.
  NFS-backed datastores) VDDK reports the whole disk as allocated and the
  full nominal size crosses the network; fastlz compresses both zero regions
  and real data on the ESXi side at low CPU cost. Measured on a 30 GiB disk:
  32.4 GB over the wire uncompressed vs 2.65 GB with fastlz.

- nbdcopy over qemu-nbd for RBD full copies: cold-direct-to-RBD and the CBT
  initial full sync to RBD previously used a single-stream qemu-img convert
  (RBD images are created inline by qemu-img, so they were never
  pre-created). They now pre-create the raw RBD image (qemu-img create),
  serve it over a localhost qemu-nbd bridge, and copy NBD-to-NBD with
  multi-connection nbdcopy - the same bridge approach the CBT cutover uses -
  falling back to qemu-img convert when nbdcopy is unavailable. The fresh
  image reads as zeros, so --destination-is-zero keeps it sparse. The Linstor
  block-device path already used nbdcopy directly; delta cycles are
  unchanged (a range copier is required there).
@andrijapanicsb andrijapanicsb changed the title KVM/VMware: warm/change-tracking migration and direct VDDK import into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes KVM/VMware: Direct VDDK import into Ceph/RBD and Linstor, plus importVm adoption of existing Ceph/RBD and Linstor root volumes (warm/change-tracking migration support for all) Jul 23, 2026

@rp- rp- left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The direct Linstor implementation of listPhysicalDisks looks fine.
BUT the new code always has the assumption that Linstor volumes are zeroed,
which is sadly not true, even more so as I'm working on getting shared disk support ready and that will only work on LVM-Thick (which doesn't guarantee zeroed devices).

So I think we need some checks for target-is-zero and if the block devices supports it,
I guess this could also be the case for other block storage providers.

The direct-VDDK and CBT initial-sync copies into a pre-created block-device
target passed --target-is-zero / --destination-is-zero unconditionally,
assuming a freshly created volume reads back as zeros. That holds for thin
providers (LVM-thin, ZFS, diskless) but NOT for thick ones (e.g. LVM-thick),
where a new volume can expose stale data from a previously deleted volume -
skipping the zero-writes would leak that stale data into the unwritten
regions of the migrated disk.

Add KVMStoragePool.isVolumeZeroInitialized(volumeName) (default false) and
implement it for Linstor via the existing LinstorUtil.resourceSupportZeroBlocks
provider check (true only for LVM_THIN / ZFS / ZFS_THIN / DISKLESS). The copy
paths now skip zeros only when it returns true; otherwise they write the zeros
(no-create into the device, but fully). This mirrors the check the staged
copyPhysicalDisk path already uses, and the default-false makes any future
block backend opt in explicitly. RBD is unaffected (qemu-img creates the image
itself, always zeroed).

Raised in review by the LINSTOR plugin maintainer (shared-disk support lands
on LVM-thick).
@andrijapanicsb andrijapanicsb changed the title KVM/VMware: Direct VDDK import into Ceph/RBD and Linstor, plus importVm adoption of existing Ceph/RBD and Linstor root volumes (warm/change-tracking migration support for all) KVM/VMware: VMware migration cold/warm into NFS)Ceph/RBD and Linstor, plus importVm adoption of existing Ceph/RBD and Linstor root volumes (warm/change-tracking migration support for all) Jul 23, 2026
@andrijapanicsb andrijapanicsb changed the title KVM/VMware: VMware migration cold/warm into NFS)Ceph/RBD and Linstor, plus importVm adoption of existing Ceph/RBD and Linstor root volumes (warm/change-tracking migration support for all) VMware→KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes Jul 23, 2026
@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

Good catch @rp- — fixed in da57d6a. Added KVMStoragePool.isVolumeZeroInitialized() (default false) so zero-skipping only happens when the backend guarantees it; Linstor implements it via the existing resourceSupportZeroBlocks provider check, so LVM-thick correctly writes zeros. Made it a generic pool capability as you suggested, so other block providers can opt in. Thanks for flagging the shared-disk / LVM-thick case.

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@blueorangutan package kvm

@blueorangutan

Copy link
Copy Markdown

@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with kvm SystemVM template(s). I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 18650

andrijapanicsb and others added 3 commits July 24, 2026 02:17
…versions

When importing a multi-disk VMware VM directly onto an RBD or Linstor pool, the
converted disks are discovered from the finalized libvirt domain, whose device
order (vda, vdb, ...) need not match the source disk order. sanitizeConvertedInstance
copied the source disk IDs onto the converted disks by list index, which swapped the
root and data disks: the VM's root volume ended up on a data disk and the guest could
not boot its OS. The identical source imported to NFS was unaffected.

Pair each converted disk to its source disk by the creation position embedded in the
pool image name (<uuid>-disk-NNN for RBD, <uuid>-dNN for Linstor) instead of by list
index, falling back to index pairing for OVF conversions whose images carry no such
position. Adds unit tests for reversed domain order, OVF fallback, and in-order cases.
Extends the KVM unmanaged-import / manage-volume adoption path so an existing
raw Linstor (DRBD) volume can be adopted as a VM ROOT or DATA volume, mirroring
the existing RBD adoption:

- LibvirtCheckVolumeCommandWrapper: Linstor added to the supported pool types
  and routed through the raw-block-device inspection path (qemu-img on the
  /dev/drbd device) instead of checkQcow2File, which rejects a raw device.
- LibvirtGetVolumesOnStorageCommandWrapper: Linstor added to the qemu-img
  supported types (data-volume listing/adoption); it inspects the local device
  path directly, no rbd: URI.
- VolumeImportUnmanageService.SUPPORTED_STORAGE_POOL_TYPES_FOR_KVM: add Linstor
  so standalone data-volume import/unmanage accepts it.
- VolumeOrchestrator.getSupportedImageFormatForCluster: RBD or Linstor => RAW.

Unit tests mirror the RBD cases (CheckVolume via /dev/drbd device path,
VolumeOrchestrator Linstor->RAW).
The generic IS_LOCKED signal comes from a host-local qemu-img file lock, which
for Linstor/DRBD does not reflect a resource attached to a running VM on another
node (unlike RBD's cluster-wide exclusive lock). Add a
KVMStoragePool.getVolumeInUseNode() capability (default null), implemented for
Linstor via the existing LinstorUtil.isResourceInUse cluster-wide InUse query,
and OR it into the IS_LOCKED detail in both the CheckVolume (root) and
GetVolumesOnStorage (data) adoption paths, so a volume still in use anywhere in
the cluster is refused with a clear reason.
@andrijapanicsb andrijapanicsb changed the title VMware→KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes VMware→KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes and Linstor root/data volumes Jul 24, 2026
@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@blueorangutan package kvm

@blueorangutan

Copy link
Copy Markdown

@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with kvm SystemVM template(s). I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 18653

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants